Electronic mail data modeling for efficient indexing

ABSTRACT

Techniques are herein described for creating a scalable IMAP4 compliant email system using a NoSQL database and a distributed full text search engine. Data for each email message is stored in multiple tables to avoid storing redundant data unnecessarily. However, a full text search index is created based on a single table as if the index refers to a single table. In embodiments herein described, the single index is created on the fields of a message metadata table with virtual fields added to the table that are derived from the message content. During this process, data is pulled from a message table in “blob” format and broken down into corresponding fields and data items, so the data items may be converted and placed in the proper virtual fields for index creation. Each blob section that is converted is cached, so the same blob section does not need to be converted multiple times. After index creation, the index may be used to search for emails based on metadata and data within the body of the email.

FIELD OF THE INVENTION

The present invention relates to email storage and retrieval techniques,and more specifically to improved computer-implemented techniques forstoring and indexing electronic mail.

BACKGROUND

There is a desire in the art to provide a scalable email store for anemail system that is searchable according to the IMAP4 specification(RFC 3501, RFC 5957, RFC 7162), all of which is incorporated herein byreference. A common email system design stores individual emails inindividual files in a file system. Metadata regarding each email may bestored in a separate metadata database. To provide searchingfunctionality, each email belonging to a particular user may bedownloaded on a client, and a text search may be performed on the clientside. Alternatively, a server side search includes using a database toidentify every email that belongs to a particular user account, and thenopens each individual message and searches the contents. Unless theclient usage of the server is limited, a client can potentially consumelarge amounts of server resources when storing and searching email.

With email in particular, certain sections of the email are likely to becopied many times. For example, an email may be sent from one user tothree other users. In that case, the three emails will have the samesender, subject, body, and attachment information. These fields maycontain a lot of data. For example, the body and attachment may be sizedanywhere from 10 MB to 50 MB of data. If a large data item must bestored each time an email is sent to a different user, then that dataitem will take up additional space each time it is copied.

Normalized data models refer to placing unique data items in separatetables, and having some tables reference the tables with the unique dataitems. Under this model, a large data item need only be stored once andother tables may reference the large data item. For example, the body ofan email could be placed in a first table and referenced by a table thattracks the different recipients of the email.

The IMAP4 specification requires that an email be searchable by manydifferent fields. Performance of a search can be sped up by indexingthese fields. But an index on multiple tables is performed less quicklybecause a search of an index may only return the primary key of onetable. Thus, if two different tables are indexed, when a search isperformed for a term in a first table AND a term in a second table, theresults from the index of the first table and the second table wouldreturn result records corresponding to different tables. Some form ofjoin operation would need to be performed to determine how the recordscorresponding to the different tables intersect. Such operations arecomputationally costly.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 is a block diagram illustrating a system architecture of an emailsystem;

FIG. 2A is a block diagram illustrating a message data table, a metadatatable, and search engine index lists;

FIG. 2B is a block diagram illustrating transforming three records intothree index documents according to an index keyspace;

FIG. 2C is a block diagram illustrating an indexed document beingcached, and then used to create two more index documents;

FIG. 2D is a block diagram illustrating caching an index document 244 asa header cache record and two sections including an attachment;

FIG. 3 is a flow chart illustrating an electronic message beingnormalized and inserted into a NoSQL database according to an exampleembodiment;

FIG. 4 is a flow chart illustrating an electronic message beingconverted into a denormalized index entry according to an exampleembodiment;

FIG. 5 is a block diagram illustrating a search for the word “big” inthe body of an electronic message and “URGENT” in the subject;

FIG. 6 is a block diagram illustrating the system architecture beingscaled up by adding a node to the distributed email system;

FIG. 7 is a block diagram illustrating a computer system that may beused to implement the techniques described herein.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

General Overview

Techniques are herein described for creating a scalable IMAP4 compliantemail system using a NoSQL database and a distributed full text searchengine. Data for each email message is stored in multiple tables toavoid storing redundant data unnecessarily. However, a full text searchindex is created based on a single table as if the index refers to asingle table. The single index is created on the fields of a metadatatable with virtual fields added to the table that are derived from themessage content. During this process, data is pulled from a messagetable in “blob” format and broken down into corresponding fields anddata items, so the data items may be converted and placed in the propervirtual fields for index creation. Some text may be standardized beforeit is indexed. Additionally, attachments may be converted into textbefore the attachments are indexed. Each section that is transformed iscached, so the same blob does not need to be converted or standardizedmultiple times even if it is sent to multiple users.

After index creation, the index may be used to search for emails basedon the fields derived from the content of the email. The indexed data iseasily queried using complex conditions because it is based on a singletable. When a person needs to perform a search based on one or morefields, the indexed columns are searched based on the conditions placedon each field, and results are returned based on one or more particularidentifying keys returned after evaluating each condition. Theparticular one or more identifying keys refer to one or more identifyingkeys in one or more identifying key columns of metadata table. A rowreferred to by the particular identifying keys has columns for messagespecific metadata, and a column that maps to the message table forobtaining a particular message in blob format.

System Architecture

Referring to FIG. 1, it is a block diagram that depicts a distributedemail system 100 according to one embodiment. In the embodimentillustrated in FIG. 1, node 102 has access to local persistent storage116, and node 122 has access to local persistent storage 136. The NoSQLdatabase 140 is spread across multiple persistent storage devices 116and 136 in a share-nothing architecture. Similarly, the search engineindex 150 is stored across multiple persistent storage devices 116 and136.

In some embodiments, the persistent storage on a single machine may be asingle physical device with storage space allocated and configured fordifferent types of data. For example NoSQL database 140-1 and searchengine index 150-1 may be located on a single storage device 116, andNoSQL database 140-2 and search engine index 150-2 may be located on asingle storage device 136. Alternatively, persistent storage may belocated on a separate machine from nodes 102 and 122. For example, NoSQLdatabase 140 and search engine index 150 may be located on one or moreseparate devices (not shown).

Nodes 102 and 122 respectively have one or more processors 104 and 124,and local volatile memory 106 and 126. In addition, nodes 102 and 122are executing database server instances 112, 132, and search enginecores 108, 128. Nodes 102 and 122 are further connected to email servers107 and 127. While in the illustrated embodiment each email server isexecuting on a separate node from the database server instances andsearch engine cores, in alternative embodiments a single node mayexecute an email server, database server instance, and search enginecore.

Email Server

In some embodiments, email servers 107, 127 are configured to send,receive, and provide access to emails according to one or morestandardized protocols. An email server 107 sends and processes messagesfor a registered network or domain according to a particular protocol.For example, an SMTP server receives an email in the formatname@domain.com. In some embodiments, other application level protocolsare used to receive electronic messages. A client may have one or moreemail accounts to receive email. A clients may access email through oneor more application level protocols. These protocol include IMAP, POP,HTTP. In alternative embodiments, a client may have one or moreelectronic contact addresses to receive electronic notifications throughone or more application level protocols. These protocols include, butare not limited to, Gadu-Gadu, IRC, Windows Messenger service, MSNP,Mumble, OSCAR, RVP, SIP/SIMPLE, Skype, TOC2, XMPP/Jingle, YMSG, andZephyr Notification service. An electronic contact address is chosenfrom a group comprising username, user ID, email address, screen name,handle, avatar, IP address, or other identifying information.

NoSQL Database Components

NoSQL database components comprise database server instances 112, 132and database partitions 140-1, 140-2. In some embodiments, cachepartitions 160-1, 160-2 may also be implemented as a NoSQL databasecomponent. The components of a NoSQL database are similar to thecomponents of a relational database except for certain functionalitythat is removed to increase performance. A typical relational databasehas each unique data item stored in a single table and that data isaccessed through a primary key. Data items from different tables may berelated through the use of a foreign key that references the primary keyof another table. Data modeling through a foreign key constraintprevents a user from adding a record into a table with a foreign key ifthere is not a corresponding record in the referenced table. By storingeach data item in a single place and only storing references to otherdata items, the data is efficiently packed.

In a NoSQL database, data is not constrained by a foreign key, but thedata may still be referenced from another table. Data inserted into aparticular table may reference data from another table, but thereference is not verified by the database before the record is inserted.The NoSQL database can guarantee that data items in a particular rowremain consistent with each other, but not that data items from separatetables remain consistent with each other, so data modelling of a singleprimary table becomes extremely important. For example, three fields areimportant in locating information regarding a particular email: useridentification, folder identification and an IMAP unique identification.All of these fields are stored together in a single table rather thanstoring data from one field in one table and data from another field ina related table to avoid data duplication. As a result, the rows in thisNoSQL table typically are very wide.

NoSQL database 140 includes two tables 142 and 144 partitionedaccordingly. Each of nodes 102 and 122 is able to access a distinctpartition of records from tables 142 and 144. Specifically, node 102 mayaccess partitions 142-1 and 144-1, and node 122 may access partitions142-2 and 144-2. In some embodiments, a node 102 may access data fromanother node 122. However, it is much faster for any given databaseserver instance to access data items of a partition that resides on thenode in which the database server instance is executing. In a preferredembodiment, the NoSQL database used to implement a NoSQL email store isCASSANDRA.

Search Engine Components

The search engine components comprise search engine cores 108, 128 andsearch engine index partitions 150-1, 150-2. The search engine cores108, 128 parse the values sought after from a specific query and comparethe values to the index 150 rather than brute force searching againstthe database tables 142, 144. In order to perform a search of aparticular search term in a particular field, the search engine cores108, 128 compare the particular search term to a particular field of theindex. Each indexed field has its own inverted index of searchable termswith additional information for a full text search. This additionalinformation may include the frequency of the term in a single document,frequency of the term in all documents, and position of the terms in thedocuments. Each search engine core 108, 128 compares the search terms tothe inverted index and determines the records with any hits and returnsresulting identifying keys of the corresponding records. The resultrecords may be returned to the client in an order based on the mosthits, based on the natural clustering order of the records, or based ona particular field within the records such as “sent date.”

FIG. 2A illustrates a full text search index 150 maintained in thedistributed email system 100. The index is first clustered by fieldnames 212 from the metadata table 144 and field names derived from themessage table 142. Each field name points to a list of terms 214 thatmay be found in each field. Each term points to a list of primary keys216 found in the metadata table 144. Finally, each primary key points tothe positions 218 of the various terms within the message. Additionalcolumns may be added to each list in order to tailor a search. Forexample, number of hits is added to the list of terms 214 so that a usercan perform a sort by relevance. Additionally, columns may includefrequency within a document. An example of an open source search engineis Solr.

Indexer Components

The DB server instance 112, 132 provides an index hook that canautomatically trigger the field input transformer 114, 134 and indexer110, 130 when data is written to the NoSQL database 140.

The indexer components 110, 130 receive data from either directly fromthe metadata table 144-1, 144-2 or from the field input transformer 114,134. The indexer components 110, 130 parse values from the data itemsreceived and store these values in a full text search index 150. Anindexer component 110 pulls records from the corresponding database140-1 and makes an entry in the index 150-1 for each term, word, orvalue parsed from a given field.

After receiving a record from the metadata table 144-1, thecorresponding indexer component 110 parses the header text of the recordentered in the NoSQL database 140-1 and creates a key-value pair foreach of the header fields. FIG. 2B is a block diagram illustratingtransforming three records into three index documents according to anindex keyspace. For example, the record with the primary key (PUID:1,PARNUM:1, FID:1, UID: 1) is used to create an index document 222. Thisindex document 222 is then added to the full text search index 150.

The indexer component 110 does not index every data item as is. Insteadthe indexer component 110 may be configured to ignore common words suchas “the” and “and”. The indexer component 110 may also employlanguage-specific stemming on the words being indexed. For example, thewords “drives”, “drove”, and “driven” may be recorded in the index underthe single concept word “drive.”

Field Input Transformer

The field input transformers may standardize data items of a particularfield for indexing. Standardization may involve removing comments andwhitespace from the content. Standardization may further includetransforming the case attributes of all words to a single case attributesuch as either lower case or upper case; adding, removing or convertingdiacritical marks or specialized symbols, words, or code from eachmessage to a more conventional or searchable format; transforming allspacing characters (returns, multiple spaces, enters) into a single typeof character (i.e. single space), tokenizing common words, symbols, orcode snippets against a dictionary, or further testing the message formalicious or renegade code that may affect the search engine ordatabase.

The field input transformers may contain a component or are operativelycoupled to a component that converts data items of a particular fieldinto plain text. Different sections of a message may requirecomputationally cheaper or more expensive converting. For example, anHTML or XML section may require a less expensive conversion than a PDFsection. Additional encryption or compression may require additionalprocessing that makes converting a particular section more expensive.

In a preferred embodiment, the field input transformers 114, 134additionally derive particular fields after reading a data item that isnot being indexed. These derived fields may be used by the search enginecore to identify items in the NoSQL database without having acorresponding field in the NoSQL database. For example, in FIG. 2B someof the fields in index keyspace 228 are not in the parent metadata table144 because these fields were derived using the field input transformer.

Examples of the derived fields include, but are not limited to, theheader information of an email such as the “subject”, “to”, “from”, and“date”, the body of the email, and any attachments of the email. Somefields may be sorted by modifying some of the values normally within thefield. For example, the field input transformer derives a sortsubjectfield by stripping off prefixes such as ‘Re:” or ‘Fwd:’ from the subjectbefore indexing.

In some embodiments, the field input transformer may be described ashaving sub-modules such as a normalization module (not shown) thatstandardizes data entries, a document converter (not shown) thatconverts different file formats into plain text, and a virtual fieldplugin (not shown) that derives fields from data within a record. Inother embodiments, the field input transformer may comprise a subset ofthese modules. In embodiments where the field input transformer does notcomprise all of the various sub-modules described above, the field inputtransformer may provide requests to sub-modules external to the fieldinput transformer to perform the various tasks assigned to thesub-modules.

Message Data Keyspace

The NoSQL database may not require that each record adhere to a strictlydefined schema. More flexible schemas mean that each record can have adifferent number of fields than the previous record. Because the recordsmay be of different length, the collection of fields may be referred toas keyspaces rather than schemas. In some embodiments, a table may bereferred to as a column family because the fields for each record maychange.

The message data keyspace 202 contains at least a message blob and aunique message identification (MID) for the primary key. The uniquemessage identification may be made unique using a timestamp. The messageblob contains the unaltered content of an email message including headerinformation, body text, and attachments. The records in this key spacemay have a very large file size with each blob being constrained to asize limit of LOMB, 20 MB, 30 MB, 40 MB, 50 MB, 100 MB, or 200 MB. Verylarge messages may be broken up into multiple records.

In addition to being stored in the initial message blob, attachments maybe extracted from the blob and stored in a separate table. This tablemay include some additional fields such as thumbnail images, documenttype, and attachment title, so these attachments may be pulled separatefrom the message table. Alternative embodiments may include extractingthe message entirely from the message table, and storing the attachmentdata in a separate table. In embodiments where the attachment data isstored in a separate table, a reference to the attachment table may bemade in the metadata table or the message table.

Metadata Keyspace

The metadata keyspace 204 contains user and mailbox metadata. Theprimary key of this keyspace may consist of a set of identifying keysincluding a permanent user identification (PUID), a partition number(PARTNUM), a folder identification such as an email folder (FID), and aunique identification for each item in the folder (UID). An email foldermay have many different implementations. For example an email foldercould be implemented through a “tag” system or a “flag” system. An emailfolder system may also be implemented through traditional folders or ahierarchal directory of folders. The email folders may be depicted tothe client as either of these systems, a combination of these systems,or none these systems. If all email is stored in a single folder, thenthe client may not perceive a folder system to exist.

The metadata keyspace 204 fields comprise at least one or more fieldsused for the primary key and at least an MID field for referencing themessage table 142. The records in the metadata keyspace 204 have asmaller size than the records in the message data keyspace 202.

As an example, in FIG. 2A, table 144 has a column MID that referencesdata from a column MID of table 142. According to one embodiment, if arecord/document from table 144 is accessed, the corresponding record intable 142 is also accessed.

In some embodiments, email messages may be annotated. Annotations aredata that may be added to an email after the email has been received bythe database system. An annotation field may be part of the metadatakeyspace 204. The annotation field may be part of the metadata table orstored in an additional table and referenced similar to the messagetable. For each record, the annotation field may contain an indicationof a row in an annotation table or an indication of the annotation textitself in the annotation field.

In some embodiments, email messages may be flagged. Flags may be used toindicate a particular email has been annotated. Additional flags may beused to mark an email as read, unread, deleted, and so forth. Theseflags may be incorporated into the metadata keyspace. For each record,the flag field(s) may contain an indication of a row in a flag table oran indication of the flag itself in the flag field.

In some embodiments, the metadata keyspace may have additional fields.The metadata keyspace may include columns according to the followingtable:

TABLE 1 Logical Name Type Description Indexed puid text Permanent uniqueYes user ID partnum bigint Partition number Yes fid timeuuid Uniquefolder ID Yes uid bigint Message UID Yes mid timeuuid Unique message IDNo (Used to fetch msg blob from message table) section blob Messagesections info. No Used to retrieve the offsets of message parts andtheir hash values to lookup the cache table. flags set<text> Set ofsystem and user Yes defined flags for the message modseq text Modseq Yesannotation map<text,blob> Map of key and Yes annotation text value sizebigint message size Yes Internaldate timestamp message arrival Yes(ARRIVAL sort timestamp - includes criteria) time and timezone and isused only for sort, and not for search. sentdate timestamp message sentYes timestamp - includes time and timezone and is used only for sort,and not for search. UTF-8 header text Header fields in UTF-8 No

In some embodiments, the metadata keyspace may include additional fieldsfor attachments. For example, rather than including attachmentinformation in the data stored in the section field, a separate“attachment position” field may be included in the metadata table. Theseadditional fields may be indexed with the metadata table in theseembodiments. In some embodiments, these additional fields are stored ina separate table and indexed separately. In such cases, multiple fieldsmay be copied from each table. For example, both tables would includePUID, PARTNUM, FID, and UID.

Full Text Search Index Keyspace

The full text search index keyspace 228 contains fields parsed from themessage blob as well as fields from the metadata keyspace 204 asindicated in table 1. A full text search index refers to an invertedindex for strings corresponding to words in a dictionary. The full textsearch index 150 is depicted in FIG. 1. FIG. 2B depicts an indexkeyspace 228 with keys for primary key fields including “PUID”,“PARTNUM”, “FID” and “UID”; header information including “sent”, “to”,“from”, and “subject”; body text; and metadata information including“flags”. The full text search index keyspace 228 comprises at least thekeys used for referencing the primary key of the metadata keyspace 204,and one key derived from the message blob such as “sent”, “to”, “from”,“subject”, “body”, or “attachment.”

The index keyspace may include fields according to table 2 in additionto the fields from columns which are indexed as described in table 1.

TABLE 2 Field Name Type Description Sent_d timestamp Sent date field -same as sentdate from table 1 except ignoring the time and timezone.This field is used for search. Internal_d timestamp Internal datefield - same as internaldate from table 1 except ignoring the time andtimezone. This field is used for search. Subject_h text Subject headerfield Sortsubject_k string Sort subject field, tokenized as a keyword -after stripping prefixes and suffixes such as Re: or Fwd: To_h email To:header field, tokenized as an email. Used for search. Sortto_k string[IMAP] addr-mailbox of the first “To” address, tokenized as a keyword.Used for sort. Displayto_k string Sort displayto field (rfc 5957),tokenized as a keyword. From_h email From: header field, tokenized as anemail. Used for search. Sortfrom_k string [IMAP] addr-mailbox of thefirst “From” address, tokenized as a keyword. Used for sort.Displayfrom_h string Sort displayfrom field (rfc 5957) Cc_h email Cc:header field, tokenized as an email. Used for search. Sortcc_k string[IMAP] addr-mailbox of the first “Cc” address, tokenized as a keyword.Used for sort. Answered_f boolean Sort answered flag field Draft_fboolean Sort draft flag field Flagged_f boolean Sort flagged flag fieldSeen_f boolean Sort seen flag field Deleted_f boolean Sort deleted flagfield *_h text Dynamic headers (fields will be dynamically generated forcustom header fields) *_f boolean Dynamic flags (fields will bedynamically generated for custom flags) *_k string Dynamic sort fieldswhich will be tokenized as keywords. Used for subject, to, cc,displayfrom and displayto sort criteria. msgtext text Search text field(header + body + attachments) msgbody text Search body field (body +attachments)

In a preferred embodiment, the index keyspace 228 may include additionalfields for attachments. These additional fields may be indexed with thefull text search index in these embodiments. In alternative embodiments,these additional fields are stored in a separate table and indexedseparately. In such cases, multiple fields may be copied from themetadata table. For example, both the metadata table and the attachmentmetadata table includes fields for PUID, PARTNUM, FID, and UID. Undereither embodiment, the index keyspace may include fields according tothe following table in addition to the fields described in table 1 andtable 2:

TABLE 3 Field Name Type Description atttype text Type derived fromindex: atpdf, atdoc, atodf, etc. attsize int Size of attachment textattname text name of attachment file attauthor text author ofpublication (if found in attachment) attpubdate text date of publication(if found in attachment) atttitle text title of publication (if found inattachment) attbstruct text body structure position attheight text imageheight info (atimage only) attwidth text image width info (atimage only)attprecision text image precision info (atimage only) attcompress textimage compression info (atimage only) attsource text image source info(atimage only) attcontent text converted text (for indexing) thumbsmallblob small thumbnail thumbmed blob medium thumbnail thumblarge bloblarge thumbnail thumbxlarge blob extra-large thumbnail

Cache Keyspace

The cache keyspace 240 contains a hash key for referencing a cachedvalue and some data that has been processed by the field inputtransformer. FIG. 1 depicts cache table 160 as being separated intothree different types of data—headers 162, bodies 164, and attachments166. These types of data are all stored in a single cache table 160, sothe FIT may acquire any cached data in a single read rather than threeseparate reads. FIG. 2D depicts a cache keyspace 240 with a record forheader information in the first row, a record for standardized body textin the second row, and a record for a converted attachment in the thirdrow. An embodiment of the cache keyspace may include columns accordingto the following table.

TABLE 4 Logical Name Data Type Description key text hash value of theattachment offset bigint Offset of attContent chunk sortsentdatetimestamp Sort sent date field sortinternaldate timestamp Sort internaldate field sortsubject string Sort subject field sortto string Sort tofield sortdisplayto string Sort displayto field (rfc 5957) sortfromstring Sort from field sortdisplayfrom string Sort displayfrom field(rfc 5957) sortcc string Sort cc field attType text STATIC type derivedfrom the attachment: atpdf, atdoc, atodf, etc. attSize int STATIC sizeof attachment text attName text STATIC name of attachment file attAuthortext STATIC author of publication (if found in attachment) attPubDatetext STATIC date of publication (if found in attachment) attTitle textSTATIC title of publication (if found in attachment) attBStruct textSTATIC body structure position attHeight text STATIC image height info(atimage only) attWidth text STATIC image width info (atimage only)attPrecision text STATIC image precision info (atimage only) attCompresstext STATIC image compression info (atimage only) attSource text STATICimage source info (atimage only) content text Contents of message partbeing cached. For attachments, this is the converted text. attThumbSmallblob STATIC small thumbnail attThumbMedium blob STATIC medium thumbnailattThumbLarge blob STATIC large thumbnail attThumbXLarge blob STATICextra-large thumbnail headermap map<text,text> Map of header key-valuepairs.

Implementation Overview

In the above mentioned keyspaces, the message data is stored in themessage table 142, and metadata is stored separately in the metadatatable 144. This means for inserting and updating records into the NoSQLdatabase less data is actually written and stored (i.e. the data isnormalized). When an email is sent to three people, only one messageblob is stored in the message table, and metadata is stored for threedifferent people in the metadata table. Similarly, when a message ismarked as read, the message isn't updated in the message table. Instead,only the metadata table is updated for the user that read the email.FIG. 3 is a flow chart illustrating an electronic message beingnormalized and inserted into a NoSQL database according to an exampleembodiment. At step 302, the distributed email system 100 receives anemail. At step 304, an email record is created in the message table byinserting the content of the email as a blob in the message table, andidentifying the message blob with a message identification (MID). Atstep 306, the message blob is parsed for keywords used to fill insection data information in the metadata table 144. For example thesection information may include:

SECTION {140} Header offset: 0 len: 1445 SHA2:<hash> Body offset: 1445len: 1168678 SHA2:<hash> subparts: 2 (1) h-off:1490 h-len:95 b-off:1585b-len:2065 charset:none enc:none SHA2:<hash> (2) h-off:3697 h-len:183b-off:3880 b-len:1166194 charset:none enc:BASE64 md5:<hash> -----------subparts: 2 (1) h-off:1630 h-len:91 b-off:1721 b-len:33 charset:us-asciienc:QP SHA2:<hash> (2) h-off:1801 h-len:90 b-off:1891 b-len:1710charset:us-ascii enc:QP SHA2:<hash> ----end---- ----end---- ----end----

The above example is a multipart message with two sub-parts. A messagemay have subparts when, for example, a message is sent both as an HTMLversion and a plain text version. A message may also have subparts whenthe message comprises body text and an attachment. In the above example,the message consists of five parts: a parent header, a first subpartheader, a first subpart body, a second subpart header, and a secondsubpart body. Most, but not all, message sections require conversionprior to indexing. Attachments that are not plain text simply requiremore expensive conversion to extract plain text for indexing.

At step 307, a particular hash function (such as MD5 or SHA2) isperformed on the corresponding sections of the message blob identifiedthrough the section fields. After section information is determined, andthe MID is created, a new record is added to the metadata table at step308. After the message is stored, an acknowledgment message may be sentback to the client at step 310.

In the index keyspace 228, the message data items and metadata dataitems are indexed as if they are in the same table. The data is repeatedin the metadata table 144 for each time the message is sent to adifferent user. Additionally, the data is repeated in the metadata table144 for each folder the message is stored in.

When a user searches for a record that meets a particular predicate, allof the conditions may be compared against an index for a single table.This hybrid approach ensures quick look up for a particular data item inthe metadata table 144 along with a particular term in the body of theemail without requiring that the message data be redundantly stored.

FIG. 4 is a flow chart illustrating an electronic message beingconverted into a denormalized index entry according to an exampleembodiment. At step 402, an indexer component 110, 130 creates a keyvalue pair for each field in the metadata table according to table 1. Atstep 404, information identified in the section field is used by thefield input transformer to parse information from the message blob. Atstep 410, the message cache is read using the hash signature todetermine if the referenced section has already been converted. At step412, if a cache hit occurs, the process flow continues to retrieving thecached copy at step 414, and the index entry is created at step 422.Otherwise, at step 412, if a cache hit does not occur, a field inputtransformer 114, 134 parses the sections and retrieves the sections fromthe message blob in the message table at step 416. At step 418, thecontent extracted from the sections is standardized into searchabletext. At step 418, the standardized content is added to the messagecache, and the index entry is created at step 422. As indicated by thearrow looping back to step 410, the process may be repeated for eachsection identified.

In a preferred embodiment, step 402 occurs after step 308. However, inan alternative embodiment, the sections are converted immediately afterstep 302, and steps 410-420 execute independently of steps 304-310.Executing at least some of steps 410 through 420 prior to having aprimary key necessary to generate an index entry may be referred toherein as priming the cache.

Inserting a Message into the Email System

The email system 100 maintains a message table 142 for inserting thecontent of each electronic message. The message table 142 is not indexeddirectly by the indexer component 110, 130. For each copy of anelectronic message sent to a different user, a record is inserted intothe metadata table 144. However, the message is only stored in themessage table 142 once. The metadata table 144 contains many metadatafields, but each metadata item in these fields includes much less datathat than the corresponding blob data stored in the message table 142.

The metadata table 144 has an index hook that triggers the field inputtransformer 114 and the creation of an index document 222 whenever arecord is added or modified in the metadata table 144. These data itemsare indexed according to the column in which they are inserted as shownin FIG. 2B. In addition to indexing most of the fields located in themetadata table 144, derived index fields are created dynamically basedon the contents of the message table 142 using the field inputtransformer 114, 134 and the section information in the metadata table144.

Thus, the index hook provides the trigger to start indexing fields froma metadata record, but particular fields cause the FIT 114, 134 toderive additional data from the message table 142 to be indexed. Forexample, the “sections” field provides the starting offset location, andlength of various sections that are contained within a particularmessage. The sections are either header sections or body sections.Headers may include message parts such as “to”, “from”, “CC”, “BCC”, and“Date”. Body sections may include message parts such as one or moremessage bodies in one or more corresponding formats and one or moreattachments in one or more corresponding formats. After the field inputtransformer 114, 134 identifies data in the section field, rather thanindexing that specific data, the field input transformer 114, 134 usesthe corresponding MID field to determine the record containing themessage in the message table, and individually indexes the data fromthese sections as if they belonged to columns with names correspondingto the particular message part.

In a preferred embodiment, the location data in the sections field iscreated when the metadata record is initially inserted in the metadatatable 144 by identifying an occurrence of a section and determining thestarting offset location and the length of the section. However,alternative embodiments could merely have an identifying message partsstep incorporated in the field input transformer called by the indexinghook of the metadata table. Thus, the metadata table 144 would not havea “sections” field, and instead each section would be parsed dynamicallyduring index creation.

Additionally, fields from the metadata table 144 do not need to hold allof the data that they refer to. Instead, they may refer to data fromother tables. For example, metadata related to annotations may be storedin an annotation table and referenced using an annotation ID. A uniqueannotation ID in the metadata table 144 may be used to index fields fromthe annotation table as if the metadata table 144 had columns from theannotation ID table. Another example includes metadata regardingattachments from an attachments table. A unique attachment ID in themetadata table may be used to index fields from the attachment table asif the metadata table has columns from the attachment table. The keypoint is that after these tables are indexed, the index only refers toone table directly—the metadata table.

Creating Index Documents

Techniques are described herein for creating an index on a metadatatable 144 with virtual fields added to the table during index creation.FIG. 2B is a block diagram illustrating transforming three records intothree index documents 222, 224, 226. A metadata table 144 is used as thebase table and additional fields derived from the body of the email areadded to the index documents 222, 224, 226 during index creation. Forexample, the metadata table 144 natively includes a “send date” fieldbut does not have a “subject” field because that would require copying alot of data from the message data when the same email is sent tomultiple users. However, during index creation, each row is indexed asif a “subject” field existed, and data is pulled from the related tablein order to fill in the necessary data item in the index document 222,224, 226.

Caching the Index Documents

During indexing, data is pulled from the message table in “section”format and converted into text suitable for an index document. Thesection is broken down into corresponding fields and data items, so thedata items may be converted and placed in the proper virtual fields forindex creation. Each part of the blob that is converted is cached andlogged by creating a hash signature for that part of the blob. FIG. 2Cis a block diagram illustrating an indexed document being cached, andthen used to create two more index documents. The same blob does notneed to be converted multiple times because the hash signature for eachconverted part of the blob will indicate that the converted data isalready stored in the cache table 160.

When the same message is sent to multiple users or when the same messageis stored in multiple folders, the FIT does not need to reparse themessage every time. Instead, the parts of the message are hashed and acache table 160 row is created for each part generated based on thesection field with the hash signature being used as the key. Wheneverthe same message is indexed more than once, the cache look up preventsthe blob from being read, parsed and converted again. Instead, themessage parts needed to create an index document are simply pulled fromthe cache table 160 and re-indexed using the new PUID, PARTNUM, FID, andUID. For example, the message with the primary key (PUID:2, PARTNUM:1,FID:1, UID:1) has subject text parsed as “urgent” and body text parsedas “this message is urgent”. These data items are cached in the cachetable 160. When the additional messages with the primary keys (PUID:3,PARTNUM:1, FID:1, UID:1) and (PUID:4, PARTNUM:1, FID:1, UID:1) need tobe indexed, the hash signature of each corresponding part is comparedagainst the cache table 160 and the index documents 234, 236 are createddirectly from the cache table 160 data.

The process of identifying a cache miss and adding a given cache entryto the cache can be performed either before or after the message isactually stored in the message data table and the metadata table. Forexample, in one embodiment, the blob sections are converted to prime thecache while the message is in the incoming mail queue, but before themessage table entry is created. This requires the email server todirectly transfer the blob to the converter and wait for conversionprior to delivery, which includes creation of the message table andmetadata table entries. This reduces the latency between messagedelivery and completion of the index as the field input transformer neednot call the converter if the cache is primed.

Converting and Caching Attachments

The message blob may include one or more attachments including worddocuments (.doc, .docx), portable document format (.pdf) documents,jpegs (.jpg), audio and video files (.mp3, .mp4), and other file typesknown in the art. Attachments are converted first to text wherepossible, and then cached in a manner similar to caching header and bodytext records.

FIG. 2D is a block diagram illustrating caching an index document 244 asa header cache record and two sections including an attachment. Aparticular metadata record 242 is parsed to create an index document244. In order to parse this particular record, the attachments need tobe converted to text prior to being indexed. Future emails may includethe same attachment. Repetitive conversions can be avoided in a similarmanner by hashing each discovered attachment, and creating an attachmentcache entry in cache table 160 for each attachment using a hashsignature of the attachment as a key in addition to creating an entryfor the header of the message, and the body of the message. The thirdrecord 248 of the cache table 160 in FIG. 2D has the converted contentof a pdf document cached. Whenever the same attachment is indexed morethan once, a cache look up prevents the data from being reconverted fromscratch. Instead, the converted attachment text is simply pulled fromthe cache table 160 and re-indexed using the new primary key (PUID,PARTNUM, FID, and UID).

The standardized body text record 246 and the converted attachmentrecord 248 are not redundant. A message may be forwarded with differentbody text, but the same attachment as the parent message. In thisparticular case, the sections for each email would be different, meaninga new body text cache entry would be created, but the attachment(s)would be the same, meaning the attachment cache record 248 would providea cache hit.

Searching the Full Text Search Index

The search engine cores 108, 128 provide a scalable mechanism forsearching the NoSQL database 140. The search engine cores 108, 128 usethe index to identify records in the NoSQL database based on theiridentifying keys—PUID, PARTNUM, FID, and UID. The resulting identifyingkeys can be used to find one or more records in the metadata table 144without requiring join operations with other tables. The records in themetadata table may be used to identify a related data item anywhere elsein the email system. For example, FIG. 5 is a block diagram illustratinga search for the word “big” in the body of an electronic message and“URGENT” in the subject. A search engine core traverses the full textsearch index by first identifying the “subject” field and the “body”field in the list of fields 212. Second, the search engine coreidentifies the “urgent” term and the “big” term in the list of terms214. Then records with common identifying keys are identified in thelist of primary keys 216. Because the primary key “1” was included inboth searches, the first record from the metadata table is returned.Finally, the search engine core may use references in the metadata tableto identify any other data for the message such as the message blob fromthe message table 142 or an attachment blob from a separate attachmenttable.

Thus, any search is able to return search results based on a singletable (the metadata table), which can then be used to find anything inthe database. This is better than performing searches across indexes ofmultiple tables because the resulting records would have to be merged,which is a processor heavy operation.

Fields from different tables can be searched to return recordscorresponding to a single table. For example, a search result for a termin the body of an email returns the primary key of a record in themetadata table 144 rather than the primary key of the message table 142even though the actual body of the email may be stored in the messagetable 142. Thus, when the search engine receives a search query withmultiple conditions such as Subject: Urgent AND flag_unread, the indexprovides resulting primary keys from the metadata table for eachcondition. For example, the indexed field for “Subject” returns primarykeys 1 and 3 from the metadata table for the search term “Urgent” andthe indexed field for “flag_unread” returns primary keys 2 and 3 for thesearch term “true”. A Boolean expression such as AND may be used toreturn the primary keys that meet both conditions. In the above example,the primary key 3 meets both conditions.

In this example, the actual value “urgent” is in the message table 142,and the actual value “true” is in the metadata table 144. However, theindex worked more efficiently by only returning primary keyscorresponding to the metadata table 142.

Updating the Index

An update to the message table 142 or metadata table 144 is notreflected as an update in the index 150. A scalable search engine doesnot function by updating old index documents. Instead, the old indexdocument is ignored and a new index document is created for each changeof state of the particular email. For example, if an email is marked asread, a flag is set in the metadata table record to indicate that themessage is read, and a corresponding new index document is created.However, the header, body text, and attachments will be the same as aprevious index document, so the cache table 160 may be used to provide asignificant boost in performance in this situation. The difference isstored in the flag field in the metadata record which has a “read” flaginstead of an “unread” flag or some token indicating as much. Similarfunctionality is provided for “deleting” an email. As such, an email maybe un-deleted simply by changing the “delete” flag.

These flags are not limited to traditional flags shown in email or eventhe IMAP specification. Any vendor or user may create their own flagsthat could be added to a particular flag column. Examples include, butare not limited to, flags indicating draft, answered, or urgency, flagsindicating importance or levels of importance, flags belonging to aparticular organization or category, flags indicating spam, flagsindicating attachments or types of attachments, flags indicating groupmessages, flags indicating email was directed solely to the PUID, flagsindicating BCC to the PUID, flags indicating that the message wasblocked or quarantined, and flags indicating that the message maycontain explicit content or viruses.

In a preferred embodiment, these flags are set as Boolean indicators intheir own columns in the metadata table 144. Alternative embodimentsinclude having the flags tokenized using a dictionary and storingdifferent tokens in a single flag field in the metadata table. Otherembodiments may have groups of flags in their own columns or tables, sothey may be set as mutually exclusive or dependent on a previous settingof a flag.

Scaling the Email System

Email system 100 may be scaled by adding additional nodes as additionalresources are needed. For example, when a threshold amount of data isstored in the NoSQL database 140, an additional node may be manually orautomatically added to the email system 100. FIG. 6 is a block diagramillustrating the system architecture being scaled up by adding a node tothe distributed email system. Node 602 comprises an additional processor604, volatile memory 606, search engine core 608, indexer component 610,NoSQL database server instance 612, field input transformer 614,persistent storage 616, NoSQL database partition 140-3 withcorresponding additional partitions for the message table 142-3 andmetadata table 144-3, search engine index partition 150-3, and cachepartition 160-3 with corresponding additional cache records for headers162-3, body texts 164-3, and attachments 166-3. In some embodiments, anadditional email server (not shown) may also be added to the system.When a node 602 is added to the email system 100, there is a linearimprovement in performance because the data has very few relations. Theadditional partitions 142-3, 244-3, 150-3, 160-3 are generated for eachtable in the additional node 602. Additional messages may be directed toand stored in the additional partitions 142-3, 244-3, 150-3, 160-3 inthe additional node 602 according to the mechanisms and implementationsdescribed herein. The NoSQL database is easily scaled by removingadditional validation mechanisms that are available in a traditionalrelational database. For example, a NoSQL database may not support theconcept of ACID transactions.

Additionally, node 102 and node 122 may be copied (not shown), and theircopies may be placed at one or more geographic locations for higheravailability of the data. Availability refers to whether data can beaccessed at any given time. High availability is assumed to be providedthrough data replication in each node. Data is considered highlyavailable if a given consumer of the data can reach the requested datafive nines of the time (0.99999).

Nodes 102, 122 and the copies of nodes 102, 122 (not shown) mayindependently store email messages and respond to search queriesregarding those messages for partition resilience. Partition-resiliencemeans that the system as whole can survive an inconsistency between datareplicas. The email system 100 favors the ability to continue processingdata even if a network partition causes communication errors betweensubsystems. For example, an email system in North America may losecommunication with replicas in Europe. In some embodiments, the systemmay maintain consistency by preventing reads and writes on one systemuntil the systems can re-sync.

When an update needs to occur, the unique data item that needs theupdate is found in the database and changed. At that time, the changepropagates to the redundant copies of the record that exist in othernodes for high availability or in case of failure.

In weak consistency models, a primary data item may be updated first,and redundant copies are updated later. In the art, this is referred toas eventual consistency. NoSQL databases usually focus on highavailability and partition tolerance over a strong consistency model. ANoSQL database ensures high availability and partition tolerance byreplicating the data across various network partitions and not enforcingstrict verifications (i.e. not using strong consistency) between networkpartitions that require data to be completely consistent with allreplicas or copies. In a preferred embodiment, the NoSQL database 140,indexes 150, and caches 160 operate under eventual consistency.

Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 7 is a block diagram that illustrates a computersystem 700 upon which an embodiment of the invention may be implemented.Computer system 700 includes a bus 702 or other communication mechanismfor communicating information, and a hardware processor 704 coupled withbus 702 for processing information. Hardware processor 704 may be, forexample, a general purpose microprocessor.

Computer system 700 also includes a main memory 706, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 702for storing information and instructions to be executed by processor704. Main memory 706 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 704. Such instructions, when stored innon-transitory storage media accessible to processor 704, rendercomputer system 700 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 700 further includes a read only memory (ROM) 708 orother static storage device coupled to bus 702 for storing staticinformation and instructions for processor 704. A storage device 710,such as a magnetic disk, optical disk, or solid-state drive is providedand coupled to bus 702 for storing information and instructions.

Computer system 700 may be coupled via bus 702 to a display 712, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 714, including alphanumeric and other keys, is coupledto bus 702 for communicating information and command selections toprocessor 704. Another type of user input device is cursor control 716,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 704 and forcontrolling cursor movement on display 712. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 700 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 700 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 700 in response to processor 704 executing one or more sequencesof one or more instructions contained in main memory 706. Suchinstructions may be read into main memory 706 from another storagemedium, such as storage device 710. Execution of the sequences ofinstructions contained in main memory 706 causes processor 704 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical disks, magnetic disks, or solid-state drives, suchas storage device 710. Volatile media includes dynamic memory, such asmain memory 706. Common forms of storage media include, for example, afloppy disk, a flexible disk, hard disk, solid-state drive, magnetictape, or any other magnetic data storage medium, a CD-ROM, any otheroptical data storage medium, any physical medium with patterns of holes,a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip orcartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 702. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 704 for execution. For example,the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 700 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 702. Bus 702 carries the data tomain memory 706, from which processor 704 retrieves and executes theinstructions. The instructions received by main memory 706 mayoptionally be stored on storage device 710 either before or afterexecution by processor 704.

Computer system 700 also includes a communication interface 718 coupledto bus 702. Communication interface 718 provides a two-way datacommunication coupling to a network link 720 that is connected to alocal network 722. For example, communication interface 718 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 718 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 718sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 720 typically provides data communication through one ormore networks to other data devices. For example, network link 720 mayprovide a connection through local network 722 to a host computer 724 orto data equipment operated by an Internet Service Provider (ISP) 726.ISP 726 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 728. Local network 722 and Internet 728 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 720and through communication interface 718, which carry the digital data toand from computer system 700, are example forms of transmission media.

Computer system 700 can send messages and receive data, includingprogram code, through the network(s), network link 720 and communicationinterface 718. In the Internet example, a server 730 might transmit arequested code for an application program through Internet 728, ISP 726,local network 722 and communication interface 718.

The received code may be executed by processor 704 as it is received,and/or stored in storage device 710, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A method comprising: storing electronic mailmessages across a plurality of tables, wherein storing electronic mailmessages, for a particular electronic mail message of said electronicmail messages, includes: storing a first row in a message store table,said first row including: a particular message id in a first message idcolumn, and a particular message blob stored in a message blob column;storing one or more particular rows in a metadata table, said one ormore particular rows including: one or more particular identifying keysin one or more identifying key columns, and the particular message id ina second message id column; for at least one given particular row ofsaid one or more particular rows in said metadata table: using theparticular message id from the second message id column to identify theparticular message id from the first message id column and saidparticular message blob; and generating a virtual field index entry, ina text search index, based on the given particular row in said metadatatable, that includes: a particular field name from text parsed from theparticular message blob, and a value based on text parsed from theparticular message blob corresponding to the particular field name;receiving a query for searching for a particular string within saidparticular field name of said electronic mail messages; in response toreceiving the query, returning the one or more particular identifyingkeys for the metadata table based, at least in part, on the virtualfield index entry; wherein storing said one or more particular rows insaid metadata table includes performing at least one of: storing adifferent row in the metadata table for each recipient of the particularelectronic mail message, or storing a different row in the metadatatable for each folder of the particular electronic mail message; whereinthe method is performed by one or more computing devices.
 2. The methodof claim 1, wherein the query further comprises searching for a secondparticular string within a second particular field name; wherein themethod further comprises: identifying a second index entry using thesecond particular string; returning the one or more particularidentifying keys for the metadata table based, at least in part, on boththe index entry and the second index entry.
 3. The method of claim 1,wherein said one or more particular rows in the metadata table furthercomprise a sections entry in a sections column indicating one or moresections within the particular message blob; wherein the givenparticular row includes, in the sections column, a respective locationof each section of the one or more sections within the particularmessage blob; wherein generating the virtual field index entry includes:using the respective location of each section in the sections column toidentify the one or more sections in the particular message blob.
 4. Themethod of claim 1, wherein storing said one or more particular rows insaid metadata table includes storing a different row in the metadatatable for each recipient/folder combination of the particular electronicmail message.
 5. The method of claim 1, wherein said one or moreparticular rows in the metadata table further comprise an indication ofa particular flag in a particular flag column for a flag placed on theparticular electronic mail message.
 6. The method of claim 1, whereinstoring said one or more particular rows in the metadata table includesstoring an indication of a particular annotation in a particularannotation column for an annotation placed on the particular electronicmail message.
 7. The method of claim 1, wherein the method furthercomprises: generating at least one particular cache entry from theparticular message blob; and for a second particular electronic mailmessage of said electronic mail messages, storing another givenparticular row of said one or more particular rows in said metadatatable with an occurrence of the same particular message id; generatingat least one additional index entry in the text search index based onthe at least one particular cache entry.
 8. The method of claim 7,wherein the particular field name, the virtual field index entry, andthe additional index entry correspond to a field name for attachments.9. The method of claim 1, wherein the method further comprises: prior tostoring the first row in said message store table, identifying from theparticular electronic mail message a plurality of sections; priming acache by generating at least one particular cache entry from aparticular section of the plurality of sections; and for the particularelectronic mail message of said electronic mail messages, generating atleast one particular index entry in the text search index based on theparticular cache entry.
 10. The method of claim 1, wherein saidplurality of tables are partitioned and stored across a plurality ofnodes in a distributed email system; the method further comprising: inresponse to receiving a threshold amount of data stored on a particularnode, adding an additional node to the distributed email system;generating additional partitions for each table in the additional node;and directing additional messages to be stored in the additionalpartitions in the additional node.
 11. One or more non-transitorystorage media storing instructions which, when executed by one or morecomputing devices, cause performance of a method comprising the stepsof: storing electronic mail messages across a plurality of tables,wherein storing electronic mail messages, for a particular electronicmail message of said electronic mail messages, includes: storing a firstrow in a message store table, said first row including: a particularmessage id in a first message id column, and a particular message blobstored in a message blob column; storing one or more particular rows ina metadata table, said one or more particular rows including: one ormore identifying keys in one or more identifying key columns, and theparticular message id in a second message id column; for at least onegiven particular row of said one or more particular rows in saidmetadata table: using the particular message id from the second messageid column to identify the particular message id from the first messageid column and said particular message blob; and generating a virtualfield index entry, in a text search index, based on the given particularrow in said metadata table, that includes: a particular field name fromtext parsed from the particular message blob, and a value based on textparsed from the particular message blob corresponding to the particularfield name; receiving a query for searching for a particular stringwithin said particular field name of said electronic mail messages; inresponse to receiving the query, returning the one or more identifyingkeys for the metadata table based, at least in part, on the virtualfield index entry; wherein storing said one or more particular rows insaid metadata table performing at least one of: storing a different rowin the metadata table for each recipient of the particular electronicmail message, or storing a different row in the metadata table for eachfolder of the particular electronic mail message.
 12. The one or morenon-transitory storage media of claim 11, wherein the query furthercomprises searching for a second particular string within a secondparticular section; wherein the one or more non-transitory storage mediaadditionally contain instructions when executed by one or more computingdevices, further causing the performance of: identifying a second indexentry using the second particular string; returning the one or moreparticular identifying keys for the metadata table based, at least inpart, on both the index entry and the second index entry.
 13. Thenon-transitory computer readable storage media of claim 11, wherein saidone or more particular rows in the metadata table further comprise asections entry in a sections column indicating one or more sectionswithin the particular message blob; wherein the given particular rowincludes, in the sections column, a respective location of each sectionof the one or more sections within the particular message blob; whereingenerating the virtual field index entry includes: using the respectivelocation of each section in the sections column to identify the one ormore sections in the particular message blob.
 14. The one or morenon-transitory storage media of claim 11, wherein storing said one ormore particular rows in said metadata table includes storing a differentrow in the metadata table for each recipient/folder combination of theparticular electronic mail message.
 15. The one or more non-transitorystorage media of claim 11, wherein said one or more particular rows inthe metadata table further comprise an indication of a particular flagin a particular flag column for a flag placed on the particularelectronic mail message.
 16. The one or more non-transitory storagemedia of claim 11, wherein storing said one or more particular rows inthe metadata table includes storing an indication of a particularannotation in a particular annotation column for an annotation placed onthe particular electronic mail message.
 17. The one or morenon-transitory storage media of claim 11, containing instructions whenexecuted by one or more computing devices, further causing theperformance of: generating at least one particular cache entry from theparticular message blob; and for a second particular electronic mailmessage of said electronic mail messages, storing another givenparticular row of said one or more particular rows in said metadatatable with an occurrence of the same particular message id; generatingat least one additional index entry in the text search index based onthe at least one particular cache entry.
 18. The one or morenon-transitory storage media of claim 17, wherein the particular fieldname, the virtual field index entry, and the additional index entrycorrespond to a field name for attachments.
 19. The one or morenon-transitory storage media of claim 11, containing instructions whenexecuted by one or more computing devices, further causing theperformance of: prior to storing the first row in said message storetable, identifying from the particular electronic mail message aplurality of sections; priming a cache by generating at least oneparticular cache entry from a particular section of the plurality ofsections; and for the particular electronic mail message of saidelectronic mail messages, generating at least one particular index entryin the text search index based on the particular cache entry.
 20. Theone or more non-transitory storage media of claim 11, wherein saidplurality of tables are partitioned and stored across a plurality ofnodes in a distributed email system; wherein the one or morenon-transitory storage media additionally contain instructions whenexecuted by one or more computing devices, further causing theperformance of: in response to receiving a threshold amount of datastored on a particular node, adding an additional node to thedistributed email system; generating additional partitions for eachtable in the additional node; and directing additional messages to bestored in the additional partitions in the additional node.